Note: All of the code for this project can be found here: https://github.com/LeaCollin0518/EDAV-FinalProject.

Introduction

For our final project, we have decided to look at our class’ Spotify data, specifically from the “Your Top Songs 2017” playlists. This is a playlist made by Spotify with each user’s 100 most listened to songs in 2017. We were curious about the listening patterns of our class and the different metrics of each song along with how these metrics are correlated.

Our team members are Hammaad Adam, Lea Collin, Ling (Kelly) He, and Danielle Su. Lea contributed to cleaning the survey data and exploring what features make songs and artists popular for this year’s EDAV class. Kelly worked on exploring general audio features of all the songs and some of the survey input data. Danielle worked on creating the initial data gathering survey and exploring trends within the class in terms of how students differ. Hammaad worked on exploring the commonalities and key differences between different students’ playlists, by songs, artists, and genre. Danielle and Kelly also created the interactive D3 component of the project. All group members contributed to writing this report.

Data Description

Spotify’s “Your Top Songs 2017” playlist was only created for users that met certain criteria. The criteria were that the user listened to 5 different artists, 30 different tracks, and 60 minutes of music from January 1st to November 1st. Note that this means all of December (and therefore likely a lot of holiday music) is not included in the data.

We collected our data using Spotify’s R API, with useful information linked here: https://www.rcharlie.com/spotifyr/. If a user gives the API a user ID and a playlist ID, the API makes it extremely easy to get different attributes for each song on the playlist. Some obvious attributes include track name, artist name, etc. More interesting features, including the features we explored most, are: genre, danceability, valence, energy, loudness, etc. A more thorough description of these features can be found under Audio Features, later on in this report.

To actually get our class’ Spotify user ID’s and playlist ID’s, we sent out a survey giving detailed instructions on how to provide this information. We also asked people to put in extra information about themselves, such as age, gender, nationality, etc., so that we could include these in our analysis.

Analysis of Data Quality

As mentioned before, the Spotify API made our data collection process extremely easy. The API did not provide any missing values for the features that we looked into. The only features that were missing values were columns such as ‘album image’, which we did not look at during our analysis. Most of our data issues came from a small number of respondents to the survey and not having as many different variables to look at as we had initially hoped for. Below, we look at the different missing data patterns.

library(extracat)
survey_data <- read_csv("StudentResponses.csv", col_names = TRUE)
## Parsed with column specification:
## cols(
##   Timestamp = col_character(),
##   Username = col_character(),
##   `Columbia Uni ID:` = col_character(),
##   `Age:` = col_integer(),
##   `Gender:` = col_character(),
##   `Nationality:` = col_character(),
##   `Location in 2017:` = col_character(),
##   `Undergraduate University:` = col_character(),
##   `How often would you say you used Spotify in 2017?` = col_integer(),
##   `Spotify User Name:` = col_character(),
##   `"Your Top Songs 2017" Playlist URI:` = col_character(),
##   `If you don't use Spotify or if you also you use other platforms to listen to music, please list what you use most frequently:` = col_character(),
##   `Should you have unique listening patterns in some way, would you agree to be mentioned by name in our project presentation to the class?` = col_character()
## )
names(survey_data) <- c("Timestamp", "email", "uni", "age", "gender", "nationality", "location2017", "undergrad", "useFrequency", "username", "playlistID", "otherPlatform", "consent")
visna(survey_data, sort = 'b')

As we can see from this plot, missing the “otherPlatform” variable (and no other variables) is the most common missing pattern in the survey data. The “otherPlatform” column was used to see if people use other music-listening platforms different from Spotify (such as Apple Music, Pandora, etc.). It is actually nice that this value is missing for so many users because it means that they are regular Spotify users. The second-most common pattern is to not have any columns missing. One person didn’t give us their Spotify user ID or their playlist ID so we were not able to include them in our dataset. For the most part, this graph shows that we were able to capture the most pertinent information from our users. Though we were not able to collect as much data as we would have liked, the data we did collect was of fairly good quality in terms of what information we wanted.

We did quite a bit of analysis using artist’s genre so we took a look at how many artists have this information available. As we see below, only 7.041667% of artists present in the dataset have missing genre information, or 169 out of 2400 rows in the dataset.

data[which(data$artist_genres == "NULL"),] <- NA
print (sum(is.na(data$artist_genres))/nrow(data))
## [1] 0.07041667
print (sum(is.na(data$artist_genres)))
## [1] 169

Main Analysis

Data Cleaning

Before we get to the analysis, it is worth briefly dicussing our data cleaning process. The respondents to the survey were instructed on how to find, copy, and paste their user and playlist ID’s from the Spotify application or Spotify webplayer. However, the actual ID’s still needed to be extracted from the strings that were copied. In order to extract the ID’s, we used R’s stringr package and Regex. Once these ID’s were extracted, it was straightforward to iterate through each ID, get the playlist features using the Spotify API, and progressively build up a dataframe with everyone’s songs. We then pulled in additional information (e.g. release date, genre) for each track, once again using the Spotoify API. The code for this process can be found on github here: https://github.com/LeaCollin0518/EDAV-FinalProject/blob/master/lea-DataCleaning.R.

Key Questions

Now that we’ve discussed our data collection and cleaning, let’s dive into the analysis. Our data lends itself to a multitude of fascinating projects, and narrowing our scope of focus was a real challenge. However, we ultimately decided to focus on the key questions that allow us to comprehensively characterize the music listening habits of our DSI class. To this end, our analysis was guided by four overarching questions:

  1. What are the distinctive features of a song that is popular within the DSI cohort?
  2. How do DSI listening habits compare with US/global trends?
  3. How similar are DSI students in their music preferences?
  4. What are the key ways in which DSI students differ from one another?

How similar are DSI students in their music preferences?

First, we’d like to examine the overlap in musical tastes between DSI students. An obvious first question to ask is: how similar are any two individuals’ playlists? One way to answer this question is by determining how many songs any two users have in common. We can summarize this in the following heatmap, where the colour of each cell represents the number of songs the two playlists have in common. For the sake of contrast, we omit the songs shared by a playlist and itself (the diagonal elements of the matrix).

# Determine which users have specific songs on their playlists

song_user <- data %>% select(Name, track_name, artist_name) %>% 
              cbind(1) %>% rename(binaryid="1") %>% 
                distinct(Name, artist_name, track_name, .keep_all = TRUE) %>%
                  spread(Name, binaryid, fill = 0) %>%
                    group_by(track_name, artist_name) %>% 
                      summarise_all(sum)

nusers = length(song_user) - 2
song_intersection <- data.frame(matrix(nrow=nusers, ncol=nusers+1))
colnames(song_intersection) = append(c("Name"), names(song_user)[-c(1,2)])
song_intersection$Name = colnames(song_intersection)[-1]

for(i in 1:nusers){
  for(j in 2:(nusers+1)){
    if(i+1==j){
      song_intersection[i,j] = 0
    }
    else{
      song_intersection[i,j] = nrow(subset(song_user, song_user[,i+2]==1 & song_user[,j+1]==1))
    }
  }
}

song_intersection = song_intersection %>% gather(key = "sharer", value = "nshared", -Name)
  
theme_heat <- theme_classic() +
  theme(axis.line = element_blank(),
        axis.ticks = element_blank(), 
        axis.text.x = element_text(angle = 60, hjust=0))

ggplot(song_intersection, aes(x = Name, y = fct_rev(sharer))) + 
  geom_tile(aes(fill = nshared/100), color = "white") + 
    coord_fixed() + theme_heat + scale_x_discrete(position = "top") +
      scale_fill_gradient2(low = "white", mid = "white", high = "darkblue", 
        name = "Songs Shared \n (% of Total)", labels = scales::percent) + 
          xlab("") + ylab("")

Each user is fairly unique: the largest degree of overlap between any two playlists is ~10%. Lea, Hammaad, and Yimin have the most songs in common, whereas Eric’s taste is almost entirely distinct.

What are the key ways in which DSI students differ from one another?

A key feature of every playlist is diversity: the range of music a student listens to. We can define diversity in a number of ways, but for the purposes of this project, we will view it in two ways: variety of artists and variety of genres. Let’s first look at aritst diversity. The two charts below illustrate the range in artist diversity between different users. The graph on the left shows the total number of artists represented in a user’s playlist; the graph on the right shows the proportion of a playlist accounted for by a user’s 10 favourite artists.

artist_freq_user <- data %>% group_by(Name, artist_name)  %>%
                      summarise(Freq = n()) %>% arrange(Name, desc(Freq))
user_total_artists <- artist_freq_user %>% summarise(nartists = n())

adiv1 <- ggplot(user_total_artists, aes(x=fct_reorder(Name, nartists), 
                                        y = nartists)) + 
          geom_bar(stat = "identity") + 
            ylab("Number of Distinct Artists") + xlab("") + 
              ggtitle('Number of Distinct \n Artists') +
              coord_flip() + theme_minimal()

user_names <- unique(artist_freq_user$Name)
user_top10_artists <- artist_freq_user[1:10,]
for(i in 2:length(user_names)){
  user_top = subset(artist_freq_user, Name == user_names[i])[1:10,]
  user_top10_artists = rbind(user_top10_artists, user_top)
}

user_top10_concentration <- user_top10_artists %>% group_by(Name)  %>%
  summarise(Sum = sum(Freq)) %>% arrange(Sum)

adiv2 <- ggplot(user_top10_concentration, aes(x = fct_reorder(Name, Sum), 
                                              y = Sum)) + 
          geom_bar(stat = "identity") + 
          theme_minimal () + xlab("") + 
            ggtitle('Proportion of Songs from \n Top 10 Artists') + 
              coord_flip() 

grid.arrange(adiv1, adiv2, nrow =1 , ncol = 2)

There is clearly a massive range in artist diversity. Users like Zach and Shadi have a large number of artists in their playlists and a small percentage accounted for by their top artists, while Andrea and Mert have just the opposite. The strong inverse relationship between these two measures of artist diversity is reflected in the following scatterplot

user_artist_concentration <- user_total_artists %>% 
                              left_join(user_top10_concentration, by="Name") %>%
                                select(Name, nartists, top10 = "Sum")

ggplot(user_artist_concentration, aes(y=nartists, x=top10)) + 
  geom_point() + ylab("Number of Distinct Artists") + 
    xlab("Proportion of Songs from Top 10 Artists")

We now turn our attention to genres. Measuring a playlist’s genre diversity is trickier than artist diversity, as each song can have multiple genre tags. In order to overcome this, we introduce the concept of spanning genres: the minimum number of distinct genres required to span a user’s entire playlist. (Note: the code to compute genre diversity is fairly complex, and is not shown here for the sake of brevity. Please see the appendix for details). The chart on the left shows the genre spans for each user. We can also measure genre diversity by the percentage of a playlist’s songs that fall into the user’s favourite genres, as shown in the chart on the right.

gdiv1 <- ggplot(user_n_genres, 
          aes(x = fct_reorder(factor(Name), ngenres), 
              y = ngenres)) + 
          geom_bar(stat = "identity") + 
          theme_minimal () + ylab("") + xlab("") +
            ggtitle('Number of Spanning \n Genres') + 
              coord_flip()

gdiv2 <- ggplot(user_n_genres, 
          aes(x = fct_reorder(factor(Name), top_genre_songs), 
              y = top_genre_songs/100)) + 
          geom_bar(stat = "identity") + 
          theme_minimal () + ylab("") + xlab("") +
            ggtitle('Proportion of Songs in \n Top Genre') + 
              scale_y_continuous(labels = scales::percent) +
              coord_flip()

grid.arrange(gdiv1, gdiv2, nrow =1 , ncol = 2)

Again, we observe that the DSI has a massive range in genre diversity. The most striking example is Tom: ~90% of his songs come from the same genre (rap)! On the other end of the spectrum, Nico and Zach have both a small percentage of songs in one genre, and a large number of spanning genres.

It is also worth investigating the difference between genre and artist diversity. Are there users who listen to a large number of artists, but a small number of genres? Or is there a strong positive relation between the number of distinct artists and number of spanning genres? We investigate these relationships with the following scatterplots.

artist_genre_concentration <- user_artist_concentration %>% inner_join(user_n_genres, by = "Name")

agdiv1 <- ggplot(artist_genre_concentration, aes(x= nartists, y = ngenres)) +
            geom_point() + theme_classic() + xlab("Number of Distinct Artists") + 
              ylab("Number of Spanning Genres")

agdiv2 <- ggplot(artist_genre_concentration, aes(y= top_genre_songs, x = top10)) + 
            geom_point() + theme_classic() + 
              ylab("% of Songs in Top Genre") + 
                xlab("% of Songs By Top 10 Artists")

grid.arrange(agdiv1, agdiv2, nrow =1 , ncol = 2)

There seems to be a positive association between number of spanning genres and number of distinct artists (left panel). This makes sense, but is not particularly surprising. The chart on the right, however, is more interesting. While there does seem to be a generally positive association, it is not very strong. For example, a large number of people have ~40% of songs in their top genre, but the proportion represented by their top 10 artist varies widely. Our hypothesis is that with more sample, we would see clearer clusters. These clusters would represent artist-genre joint diversity: whether a user listens to a lot of artists across a bunch of different genres (high diversity) or a large number of artists across a small number of genres (medium diversity), or a small number of artists across a small number of genres (low diversity).

class_df <- read_csv('Final_DSI_Playlist.csv')
## Parsed with column specification:
## cols(
##   .default = col_character(),
##   playlist_num_tracks = col_integer(),
##   track_added_at = col_datetime(format = ""),
##   track_popularity = col_integer(),
##   danceability = col_double(),
##   energy = col_double(),
##   loudness = col_double(),
##   speechiness = col_double(),
##   acousticness = col_double(),
##   instrumentalness = col_double(),
##   liveness = col_double(),
##   valence = col_double(),
##   tempo = col_double(),
##   duration_ms = col_double(),
##   time_signature = col_integer(),
##   age = col_integer(),
##   useFrequency = col_integer(),
##   artist_popularity = col_integer(),
##   artist_num_followers = col_integer(),
##   rank = col_integer(),
##   album_date = col_date(format = "")
##   # ... with 2 more columns
## )
## See spec(...) for full column specifications.

In examining the differences between students in our DSI cohort, we wanted to see who listens to the most obscure music, or the most “indie”, and who listens to the most popular music, or the most “mainstream”. The best variable to do this with is Spotify’s track popularity variable, this is a number assigned from 0 to 100 that is calculated based on an algorithm that takes the total number of streams for that track and weighs more recent streams more heavily. The exact formula for this number is unknown, but it is our best measure in the dataset for how popular a song is. One caveat to this variable is that we pulled the track popularity when we built our dataset, and these playlists are from 2017, so the track_popularity is indicative of how popular that song is today rather than in 2017 when the student was listening to it. Therefore the results could also be interpreted as, who listens to the music that stays popular throughout a year’s worth of time.

We first look at this variable in conjunction with artist popularity, another possible indicator someone’s music tastes being “mainstream” or “indie”. The Cleveland dot plot below averages the track and artist popularity for each DSIers playlists. The artist popularity metric is formed similarly to the track popularity metric by Spotify. The plot reveals that Lea and Hammaad both listen to the most popular music (which is not too surprising since they have the most songs in common) and Eric listens to the leas popular music (again not surprising since he had the least number of songs in common with anyone else). An interesting thing to note from the graph is that Tom has a lower track popularity but one of the highest artist popularities.

avg_class_data <- display_class_data %>% group_by(Name) %>% summarise_at(vars(normalized_artist_pop, normalized_pop, danceability, energy, valence, normalized_loudness, speechiness, acousticness, normalized_instrumentalness, normalized_tempo), funs(mean(., na.rm = TRUE)))

names(avg_class_data) <- c("Name", "Artist Popularity", "Track Popularity", "Danceability", "Energy", "Valence", "Loudness"," Speechiness", "Acousticness", "Instrumentalness", "Tempo")

tidy_avg <- avg_class_data %>% select(Name, `Track Popularity`, `Artist Popularity`) %>% 
  arrange(`Track Popularity`) %>% 
  mutate(Name = factor(Name, levels = .$Name)) %>% tidyr::gather(key = 'Category', value = 'Percentage', -Name)

pop_cd <- ggplot(tidy_avg, aes(x = Percentage, y = Name, color = Category)) +
          geom_point() + ylab("") + theme_minimal() +
          ggtitle("Hammaad and Lea Are Basic AF") +
        scale_color_manual(values=c("red", "blue")) +
    theme(legend.title=element_text(size=15), legend.text=element_text(size=13))

pop_cd

Below we created 2 sets of boxplots - they are boxplots of the distribution of their playlist’s popularity variables for each student in our dataset. You can see based on the first set which has track popularity that Lea listens to the most popular music in terms of having the highest median track popularity value and Eric has the lowest median track popularity. Additionally, from the boxplots you can see that some students such as Mert and Anonymous 1 have very large ranges whereas James and Hammaad’s ranges are rather small in comparison.

Additionally we made the same plot for the artist popularity variable, ordered in the same way as track popularity, so it is easy to compare. From Spotify’s description, the artist popularity variable is determined by the track popularity so it should be a similar graph. However, when you compare, you can see that all ranges are smaller and in general they are on the higher end of the popularity variable. This could be explained by a few reasons, maybe artist popularity distributions in general are higher, or students could be listening to low popularity tracks but from artists that do have very high popularity tracks, or it could be a factor of the time difference. Perhaps the tracks that students listened to in 2017 are no longer very popular, but the artist has new tracks in 2018 that have brought the artist’s popularity variable up.

mytheme <- theme_minimal(16) +
    theme(plot.title = element_text(hjust = 0.5))

# Boxplots of track popularity for each student, ordere by highest median track popularity to lowest
ggplot(class_df, aes(x=fct_reorder(Name, track_popularity), y=class_df$track_popularity)) +
  geom_boxplot(fill="#CCCCFF") +
  scale_x_discrete("") +
  labs(y="Track Popularity", x="DSI-er", title="Who is the most indie and the most mainstream") +
  coord_flip() +
  mytheme

# Boxplots ordered the same as track_popularity (for comparison) but showing artist popularity
ggplot(class_df, aes(fct_reorder(Name, track_popularity), artist_popularity)) +
  geom_boxplot(fill="#CCCCFF") +
  labs(y="Artist Popularity", x="DSI-er", title="Does this look the same when showing artist popularity?") +
  coord_flip() +
  mytheme
## Warning: Removed 4 rows containing non-finite values (stat_boxplot).

The next few graphs try to use the survey inputs to identify any trends within those survey answers. While the number of tracks in our dataset is sizeable (2400), they only come from 24 students, and when separating that into categories based on survey answers, it is hard to really identify any trends. We were mainly interested in the gender variable and the age variable. For gender, the distribution in the dataset is 8 females and 16 males. For age, the distribution in the dataset is age 21: 1, age 22: 4, age 23: 4, age 24: 5, age 25: 5, age 26: 2, age 28: 2. Therefore while the graphs using age are still interesting, it is hard to note any general trends, since if you’re talking about 21 year olds it is really just that 1 specific person. Additionally the ages are all in the 21-28 range, so we wouldn’t expect a huge amount of difference in their listening habits, but you can see some distinctions.

Below we made a mosaic plot of age and track popularity, which is binned into 4 quartiles since the original number is likely a percentile. You can see from the mosaic plot that there isn’t much of a relationship between age and listening to popular or not popular tracks. It looks like 22 year olds listen to the most popular tracks, but this could also be because there is only 1 21 year old and at least 4 22 year olds. Therefore, nothing conculsive can really be said about age and popularity. If given more time, we could collect a wider range of samples and test this again.

library(forcats)
library(ggmosaic)
## 
## Attaching package: 'ggmosaic'
## The following object is masked from 'package:GGally':
## 
##     happy
popular_df <- class_df %>% select(track_popularity, age)

# Binning the track popularity into quartiles
df <- popular_df %>% select(track_popularity, age) 
df$track_popularity[df$track_popularity < 25] <- "[1-25)"
df$track_popularity[df$track_popularity >= 25 & df$track_popularity < 50] <- "[25-50)"
df$track_popularity[df$track_popularity >= 50 & df$track_popularity < 75] <- "[50-75)"
df$track_popularity[df$track_popularity >= 75] <- "[75-100)"

df$age <- factor(df$age)
df$track_popularity <- factor(df$track_popularity)

ggplot(df) +
  geom_mosaic(aes(x = product(age), fill=track_popularity), na.rm=FALSE)

Next, we wanted to explore the album data we were able to pull and see if there are any trends based on the year the album was released and therefore when the song was released.

First, we explored album dates in relation to gender of the student. Below is a bar graph of the number of plays of the track and the decade the song was released, faceted by gender. You can see that the distribution looks similar but there is a little cluster of songs released in the 70s for the males - classic rock songs, no doubt. Since there are a lot more males than females in our dataset, we graphed the same thing again in terms of percentages. In that next graph you can now see that the females in our group have over 80% of their songs coming out of the 2010s decade. So it seems like everyone likes more recent music, but especially DSI females!

# Getting total Male/Female counts to calculate percentages
gender_count <- class_df %>% group_by(gender) %>% summarize(total=n())
female_count <- as.integer(gender_count[1,2])
male_count <- as.integer(gender_count[2,2])

# Creating new dataframe with grouped values by decade and gender
class_df_grouped <- class_df %>% group_by(album_decade, gender) %>% summarize(Freq=n()) %>% mutate(Percent= ifelse(gender=='Female', 100*(Freq/female_count), ifelse(gender=='Male', 100*(Freq/male_count), 0)))

# Bar chart with counts
ggplot(class_df_grouped, aes(x=album_decade, y=Freq)) +
  geom_histogram(fill="thistle", stat="identity") +
  facet_wrap(~gender)
## Warning: Ignoring unknown parameters: binwidth, bins, pad
## Warning: Removed 2 rows containing missing values (position_stack).

# Bar char with percentages - you can really see the percentage of females listening to recent music is high
ggplot(class_df_grouped, aes(x=album_decade, y=Percent)) +
  geom_histogram(fill="thistle", stat="identity") +
  facet_wrap(~gender) +
  labs(xlab='Decade of Album Release', title='Everyone likes more recent music, but especially DSI Females!')
## Warning: Ignoring unknown parameters: binwidth, bins, pad

## Warning: Removed 2 rows containing missing values (position_stack).

Since the majority of the songs came out of alubms in the 2000s we wanted to do a deeper dive into the 2000s to see if there are any trends within years of the 2000s.

In terms of gender, the distributions of the 2000s look similarly skewed. However, males have fatter tails in terms of having a higher percentage from 2000 to 2012, so while they are listening to more recent music, maybe they like to listen to throwback songs a little more than females. Additionally, now that we are zooming into the 2000s you can see that of the 2010s the majority of the songs are from 2016-2017, so either songs that came out the year of the playlist or songs that came out the year right before it.

We would also like to note that there are some album release dates in 2018, even though this was a playlist of top songs played in 2017. This is due to the fact that the way Spotify has tracked their album release dates, it was easiest to take the release date from the actual album, whereas it might have shown up on the playlist because the single came out in 2017, but the album did not come out until 2018.

# Deeper Dive into the 2000s
year2000 <- class_df %>% filter(album_year >= 2000)

# Creating new dataframe with grouped values by year and gender
year2000_grouped <- year2000 %>% group_by(album_year, gender) %>% summarize(Freq=n()) %>% mutate(Percent= ifelse(gender=='Female', 100*(Freq/female_count), ifelse(gender=='Male', 100*(Freq/male_count), 0)))

# Male/Female distributions are pretty similar
ggplot(year2000_grouped, aes(x=album_year, y=Percent)) +
  geom_histogram(fill="thistle", stat="identity") +
  facet_wrap(~gender) +
  labs(xlab='Year of Album Release', title='Similar distributions, but males like to throwback a little more')
## Warning: Ignoring unknown parameters: binwidth, bins, pad

Since the gender distributions were similar, even though age cannot be generalized, we wanted to see if age had any trends with album release year. The below graphs are faceted by age and you can see that generally as you increase in age, the skew to 2016-2017 is less and less extreme and students listen to more and more songs released in the early 2000s. In order to better compare, we made a plot with density curves of each age. Here you can see similar trends, but you can really see that 26 year olds have the lowest density around 2016-2017 and are much more evenly spread out than the others. You can also see that 21 and 22 year olds have the lowest percentages of songs from 2000 to about 2007.

# Creating new dataframe with grouped values by year and gender
age21 <- length(which(class_df$age == '21'))
age22 <- length(which(class_df$age == '22'))
age23 <- length(which(class_df$age == '23'))
age24 <- length(which(class_df$age == '24'))
age25 <- length(which(class_df$age == '25'))
age26 <- length(which(class_df$age == '26'))
age27 <- length(which(class_df$age == '27'))
age28 <- length(which(class_df$age == '28'))

year2000_grouped <- year2000 %>% group_by(album_year, age) %>% summarize(Freq=n()) %>% 
  mutate(Percent= ifelse(age=='21', 100*(Freq/age21), ifelse(age=='22', 100*(Freq/age22), ifelse(age=='23', 100*(Freq/age23), ifelse(age=='24', 100*(Freq/age24), ifelse(age=='25', 100*(Freq/age25), ifelse(age=='26', 100*(Freq/age26),ifelse(age=='27', 100*(Freq/age27),ifelse(age=='28', 100*(Freq/age28), 0)))))))))

# Bar char with percentages
ggplot(na.omit(year2000_grouped), aes(x=album_year, y=Percent)) +
  geom_histogram(fill="thistle", stat="identity") +
  facet_wrap(~age) +
  labs(xlab='Year of Album Release', title='Year')
## Warning: Ignoring unknown parameters: binwidth, bins, pad

# Density curves
year2000clean <- na.omit(subset(year2000, select=c(album_year, age)))
Student_Age <- as.factor(year2000clean$age)
ggplot(year2000clean, aes(x = album_year, color=Student_Age)) +
  geom_density() +
  ggtitle("Age/Year of album distribution") +
  labs(x = "Year of Album Release", y = "Density") +
  theme(plot.title = element_text(hjust = 0.5))

We were curious to see how all these factors: age, track popularity, and album release date come together, so we plotted them on a scatterplot with date on the x-axis and popularity on the y-axis. The points are colored by the age of the student. In accordance with the previous plots, most of the songs are clustered around 2016-2017, and they range the full spectrum of track popularity with a big cluster in the 30-75 range and a small clusterin the 0-15 range. Generally, it looks like if younger students listen to older songs they at least have a track popularity of 40 or above. The only students that listen to tracks released before 1991 and are below 40 in popularity are 24 and 25 year olds. You can see this even better in the interactive and subsetted graph described below.

In order to see specifics of these outliers, and to better visualize the graph, we subset the data so it only includes the top 25 songs in each person’s playlist, presumably these are their top 25 favorite songs of 2017. Then, we used plotly and displayed the track name and artist name so we could see what exactly these points represented. The oldest song in this subset is “I Fall in Love too Easily” by Chet Baker with a popularity level of 62. The highest track popularity for a song that came out before 2000 is 81 and that is for “Under the Bridge” by Red Hot Chili Peppers. The highest track popularity overall is 89 and is the song Jocelyn Flores by XXXTENTACION. The lowest track popularity for a song that came out before 2000 is 0 and is the song “Come on Home” by Joan Armatrading.

splot_theme <- theme_classic(12) + theme(plot.title = element_text(hjust = .5))
library(viridis)
## Loading required package: viridisLite
library(plotly)
## 
## Attaching package: 'plotly'
## The following object is masked from 'package:ggplot2':
## 
##     last_plot
## The following object is masked from 'package:stats':
## 
##     filter
## The following object is masked from 'package:graphics':
## 
##     layout
# Scatterplot with all songs comparing album release date and track popularity, colored by age
class_df_clean <- na.omit(subset(class_df, select=c(album_date, age, track_popularity)))
Age <- as.factor(class_df_clean$age)
ggplot(class_df_clean, aes(album_date, track_popularity)) + 
    geom_point(aes(color= Age), alpha = .7, size = 2, stroke=0) +
    scale_color_viridis_d() +
    splot_theme + 
    labs(title= 'How does album release date relate to track popularity?', subtitle = "All Tracks",
         x = 'Album Release Date', y = 'Track Popularity Levels')

# Same as above, but with only the top 25 songs per student's playlist
top50_df <- na.omit(subset(class_df, rank <=25, select=c(album_date, age, track_popularity, track_name, artist_name)))
Age <- as.factor(top50_df$age)
g<- ggplot(top50_df, aes(album_date, track_popularity, text=paste("Song:",track_name, "Artist:",artist_name))) + 
    geom_point(aes(color = Age), alpha = .7, size = 2, stroke=0) +
    scale_color_viridis_d() +
    splot_theme + 
    labs(title= 'How does album release date relate to track popularity?', subtitle = "Top 50 Tracks of Each Playlist",
         x = 'Album Release Date', y = 'Track Popularity Levels')

# Made interactive so we can see what those outlier songs are that were released a long time ago but have high track popularity
# Hover over points to see what song it is and the artist in addition to date of release and track pouplarity
ggplotly(g)

Interactive Component

Link to the Interactive Component: https://bl.ocks.org/daniellesu/bda312bb5a71159d3bc7483c278ddd96/070d7b9a58c3813c5c624997856a1e88376faeb2 Hosted on Block Builder

For our interactive component, we decided to make a game where the users can gain general knowledge about the audio features of a song as well as interesting trends of popular songs in DSI 2017 playlists. We used D3 to build this interactive component.
Specifically, we made a DSI 2017 Spotify playlist with the top 10 songs out of all 2400 songs we collected from users. In this game, a user is asked to play a song from the DSI 2017 playlist, select the song from the dropdown list, and guess the values of eight audio features by dragging the bars (between 0.0 and 1.0) in a bar plot where each bar corresponds to one audio feature. Once the user finishes guessing, s/he can click “Submit”, and the actual values and the median values (of all 2400 songs) of those audio features will display in the bar plot. The user can then click “Play Again”, select another song from the dropdown list, and repeat the guessing process.

The instructions/rules of the game and the audio features are explained on the webpage, so that the users can easily navigate through the page and play the game. We also made some adjustments to the data to facilitate this game. Particularly, we normalized the values of three audio features, loudness, tempo, and track popularity, so that all the values fall between 0.0 and 1.0 (inclusive), and the bars of all audio features are on the same scale.

There are also several things that we attempted and would like to improve in the future. A small caveat in the technical execution is that when the user drags the bar up and down, if the bar is dragged below 0.0 by accident and the user lets go of the click, the user wouldn’t be able to drag it back up. However, you are able to click “Play Again” to refresh the screen and get all the bars back. Another issue is if you keep clicking “Submit” over and over again, it will keep appending new bars with the actual values - if we had more time we would edit this so that wouldn’t happen. Additionally, if you do not select a song and you click submit, the bars will will fade, in the future we would add a warning that says you have not selected a song and would not let the bars change. If we had more time, we would try to display the numbers of the actual values and the median above the bars. It would also be interesting to calculate a metric that describes how close a user’s guesses are to the actual values, indicating how strong the user’s music sense is and display that as a result.

We also considered universal design when we created this interactive component. We wanted to make sure that our game is accessible to users with color vision deficiency (CVD). Therefore, we used Color Oracle, a color deficiency simulator, to test our interactive component on all 3 types of color deficiency. In the end, we selected the colors of the bars in such a way that the bars with actual values are still differentiable from the colors of the bars with a user’s guesses. Furthermore, we have the transition of the bars with actual values go from bottom to top so that the users can see them coming up and see where they are stopping in relation to the bars with their guesses. With this design, users with CVD would have no problem playing our game. However, we could not make this game accessible to users with hearing impairment, since it asks the user to guess values after listening to a song. In the future, we would like to find an alternative way for those users to enjoy the game.

Conclusion

The main limitation of our project is that we didn’t collect as much data from our classmates as we would have liked. We ended up collecting playlist information from 24 users (including ourselves). We attribute the lack of responses in part to the fact that Spotify was not accessible to everyone in 2017. For example, Spotify is not supported in China. This was not absolutely detrimental to our project, however, because there are still 100 songs per playlist and people’s music taste varied enough that we had a large number of songs to analyze.

Given more time, we would have also analyzed Spotify users’ “Your Wrapped 2018”, which is the exact same concept as the “Your Top Songs 2017” playlist. We did not do this as the 2018 playlist was recently released. We wanted to do this to be able to analyze if people’s tastes had changed over time and if so, in what ways.

Through this project, we learned a lot about how to collect data through an API and use it in R. We also learned how to combine data from different sources as we had to combine the survey data with what we could get from the API. We learned more about our cohort and their propensity for loud music.